<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>PyMOTW: Queue &mdash; PyMOTW v1.2 documentation</title>
    <link rel="stylesheet" href="../_static/default.css" type="text/css" />
    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../',
        VERSION:     '1.2',
        COLLAPSE_MODINDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="../_static/jquery.js"></script>
    <script type="text/javascript" src="../_static/doctools.js"></script>
    <link rel="author" title="关于这些文档" href="../about.html" />
    <link rel="index" title="索引" href="../genindex.html" />
    <link rel="search" title="搜索" href="../search.html" />
    <link rel="copyright" title="Copyright" href="../copyright.html" />
    <link rel="top" title="PyMOTW v1.2 documentation" href="../index.html" />
    <link rel="next" title="PyMOTW: StringIO and cStringIO" href="StringIO.html" />
    <link rel="prev" title="PyMOTW: ConfigParser" href="configparser.html" /> 
  </head>
  <body>
    <div class="related">
      <h3>导航</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="一般索引"
             accesskey="I">索引</a></li>
        <li class="right" >
          <a href="StringIO.html" title="PyMOTW: StringIO and cStringIO"
             accesskey="N">后一篇</a> |</li>
        <li class="right" >
          <a href="configparser.html" title="PyMOTW: ConfigParser"
             accesskey="P">前一篇</a> |</li>
        <li><a href="../index.html">PyMOTW v1.2 documentation</a> &raquo;</li> 
      </ul>
    </div>  
    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body">
            
  
  <div class="section" id="pymotw-queue">
<h1>PyMOTW: Queue<a class="headerlink" href="#pymotw-queue" title="Permalink to this headline">¶</a></h1>
<ul class="simple">
<li>模块：Queue</li>
<li>目的：提供一个线程安全的FIFO功能.</li>
<li>python版本：1.4+</li>
</ul>
<div class="section" id="id1">
<h2>描述<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
<p>Queue提供了FIFO功能, 一般常用于多线程编程, 它可以在生产者和消费者线程中安全的传递消息或者其他数据. 调用者会自动创建锁, 当使用Queue对象, 你可以根据需求创建多个线程. 一个Queue的大小(元素的个数)受可用内存的限制.</p>
<p>本文假设你已经了解基本的Queue特点, 如果你还不清楚, 可以阅读参考后继续后面内容:</p>
<ul class="simple">
<li><a class="reference external" href="http://en.wikipedia.org/wiki/Queue_%28data_structure">Queue data structures</a></li>
<li><a class="reference external" href="http://en.wikipedia.org/wiki/FIFO">FIFO</a></li>
</ul>
</div>
<div class="section" id="id2">
<h2>示例<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h2>
<p>举例说明如何在多线程中使用Queue对象, 我们创建一个简单的 <a class="reference external" href="http://en.wikipedia.org/wiki/Podcasting">podcasting</a> 客户端, 这个客户端读取一个或者多个RSS feeds, 依次将需下载的内容置于队列中, 然后采用多线程模式同时处理多个下载. 这比较简单, 也许没有多大实用价值, 但这个框架代码很好的说明了如何来利用Queue模块.</p>
<p>开始, 我们加载一些有用的模块:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">Queue</span> <span class="kn">import</span> <span class="n">Queue</span>

<span class="kn">from</span> <span class="nn">threading</span> <span class="kn">import</span> <span class="n">Thread</span>
<span class="kn">import</span> <span class="nn">time</span>

<span class="kn">import</span> <span class="nn">feedparser</span>
</pre></div>
</div>
<p>首先, 需要创建一些运行参数, 通常这些来自用户输入(可以任何东西, 比如参数, 数据库), 在我们的例子中, 我们硬编码几个值.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># Set up some global variables</span>
<span class="n">num_fetch_threads</span> <span class="o">=</span> <span class="mf">2</span>
<span class="n">enclosure_queue</span> <span class="o">=</span> <span class="n">Queue</span><span class="p">()</span>

<span class="c"># A real app wouldn&#39;t use hard-coded data...</span>
<span class="n">feed_urls</span> <span class="o">=</span> <span class="p">[</span> <span class="s">&#39;http://www.castsampler.com/cast/feed/rss/guest&#39;</span><span class="p">,]</span>
</pre></div>
</div>
<p>接下来, 我们需要在工作线程中定义相应函数来处理下载. 再次, 这里为了便于说明模拟下载, 实际下载可以参考 <a class="reference external" href="http://docs.python.org/lib/module-urllib.html">urllib</a> 模块(这再以后会介绍). 在这个示例中, 我们只根据线程id, 使其sleep一段时间.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="k">def</span> <span class="nf">downloadEnclosures</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">q</span><span class="p">):</span>
    <span class="sd">&quot;&quot;&quot;This is the worker thread function.</span>
<span class="sd">    It processes items in the queue one after another.</span>
<span class="sd">    These daemon threads go into an infinite loop,</span>
<span class="sd">    and only exit when the main thread ends.</span>
<span class="sd">    &quot;&quot;&quot;</span>

    <span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
        <span class="k">print</span> <span class="s">&#39;</span><span class="si">%s</span><span class="s">: Looking for the next enclosure&#39;</span> <span class="o">%</span> <span class="n">i</span>
        <span class="n">url</span> <span class="o">=</span> <span class="n">q</span><span class="o">.</span><span class="n">get</span><span class="p">()</span>
        <span class="k">print</span> <span class="s">&#39;</span><span class="si">%s</span><span class="s">: Downloading:&#39;</span> <span class="o">%</span> <span class="n">i</span><span class="p">,</span> <span class="n">url</span>
        <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">i</span> <span class="o">+</span> <span class="mf">2</span><span class="p">)</span> <span class="c"># instead of really downloading the URL, we just pretend</span>

    <span class="n">q</span><span class="o">.</span><span class="n">task_done</span><span class="p">()</span>
</pre></div>
</div>
<p>一旦定义好目标函数, 我们就可以启动工作线程. 注意, 函数downloadEnclosures()在&#8221;url = q.get()&#8221;会阻塞, 直到队列有东西返回, 因此, 当队列中有东西时, 启动线程总是安全的.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># Set up some threads to fetch the enclosures</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">num_fetch_threads</span><span class="p">):</span>
    <span class="n">worker</span> <span class="o">=</span> <span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">downloadEnclosures</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">enclosure_queue</span><span class="p">,))</span>
    <span class="n">worker</span><span class="o">.</span><span class="n">setDaemon</span><span class="p">(</span><span class="bp">True</span><span class="p">)</span>
    <span class="n">worker</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
</pre></div>
</div>
<p>现在, 我们开始检索feed的内容(使用Mark Pilgrim的 <a class="reference external" href="http://www.feedparser.org/">feedparser</a> 模块)和一个url集合. 当第一个url添加到队列后, 一个工作线程即可选择它并启动下载. 循环将继续运行并添加相应的feed, 直到全部加完, 工作线程将轮流取出url去下载它们.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># Download the feed(s) and put the enclosure URLs into the queue.</span>
<span class="k">for</span> <span class="n">url</span> <span class="ow">in</span> <span class="n">feed_urls</span><span class="p">:</span>
    <span class="n">response</span> <span class="o">=</span> <span class="n">feedparser</span><span class="o">.</span><span class="n">parse</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">agent</span><span class="o">=</span><span class="s">&#39;fetch_podcasts.py&#39;</span><span class="p">)</span>
    <span class="k">for</span> <span class="n">entry</span> <span class="ow">in</span> <span class="n">response</span><span class="p">[</span><span class="s">&#39;entries&#39;</span><span class="p">]:</span>
        <span class="k">for</span> <span class="n">enclosure</span> <span class="ow">in</span> <span class="n">entry</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">&#39;enclosures&#39;</span><span class="p">,</span> <span class="p">[]):</span>
            <span class="k">print</span> <span class="s">&#39;Queuing:&#39;</span><span class="p">,</span> <span class="n">enclosure</span><span class="p">[</span><span class="s">&#39;url&#39;</span><span class="p">]</span>
            <span class="n">enclosure_queue</span><span class="o">.</span><span class="n">put</span><span class="p">(</span><span class="n">enclosure</span><span class="p">[</span><span class="s">&#39;url&#39;</span><span class="p">])</span>
</pre></div>
</div>
<p>剩下就可以等待队列为空.</p>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># Now wait for the queue to be empty, indicating that we have</span>
<span class="c"># processed all of the downloads.</span>
<span class="k">print</span> <span class="s">&#39;*** Main thread waiting&#39;</span>
<span class="n">enclosure_queue</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="k">print</span> <span class="s">&#39;*** Done&#39;</span>
</pre></div>
</div>
<p>下载如下 <a class="reference external" href="http://www.doughellmann.com/PyMOTW/fetch_podcasts.py">示例代码</a> , 运行即可看到如下输出:</p>
<div class="highlight-python"><pre>0: Looking for the next enclosure
1: Looking for the next enclosure
Queuing: http://http.earthcache.net/htc-01.media.globix.net/COMP009996MOD1/Danny_Meyer.mp3
Queuing: http://feeds.feedburner.com/~r/drmoldawer/~5/104445110/moldawerinthemorning_show34_032607.mp3
Queuing: http://www.podtrac.com/pts/redirect.mp3/twit.cachefly.net/MBW-036.mp3
Queuing: http://media1.podtech.net/media/2007/04/PID_010848/Podtech_calacaniscast22_ipod.mp4
Queuing: http://media1.podtech.net/media/2007/03/PID_010592/Podtech_SXSW_KentBrewster_ipod.mp4
Queuing: http://media1.podtech.net/media/2007/02/PID_010171/Podtech_IDM_ChrisOBrien2.mp3
Queuing: http://feeds.feedburner.com/~r/drmoldawer/~5/96188661/moldawerinthemorning_show30_022607.mp3
*** Main thread waiting
0: Downloading: http://http.earthcache.net/htc-01.media.globix.net/COMP009996MOD1/Danny_Meyer.mp3
1: Downloading: http://feeds.feedburner.com/~r/drmoldawer/~5/104445110/moldawerinthemorning_show34_032607.mp3
0: Looking for the next enclosure
0: Downloading: http://www.podtrac.com/pts/redirect.mp3/twit.cachefly.net/MBW-036.mp3
1: Looking for the next enclosure
1: Downloading: http://media1.podtech.net/media/2007/04/PID_010848/Podtech_calacaniscast22_ipod.mp4
0: Looking for the next enclosure
0: Downloading: http://media1.podtech.net/media/2007/03/PID_010592/Podtech_SXSW_KentBrewster_ipod.mp4
0: Looking for the next enclosure
0: Downloading: http://media1.podtech.net/media/2007/02/PID_010171/Podtech_IDM_ChrisOBrien2.mp3
1: Downloading: http://feeds.feedburner.com/~r/drmoldawer/~5/96188661/moldawerinthemorning_show30_022607.mp3
1: Looking for the next enclosure
*** Done</pre>
</div>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar">
        <div class="sphinxsidebarwrapper">
            <h3><a href="../index.html">目录</a></h3>
            <ul>
<li><a class="reference external" href="">PyMOTW: Queue</a><ul>
<li><a class="reference external" href="#id1">描述</a></li>
<li><a class="reference external" href="#id2">示例</a></li>
</ul>
</li>
</ul>

            <h4>前一个主题</h4>
            <p class="topless"><a href="configparser.html" title="前一章节">PyMOTW: ConfigParser</a></p>
            <h4>后一个主题</h4>
            <p class="topless"><a href="StringIO.html" title="后一章节">PyMOTW: StringIO and cStringIO</a></p>
            <h3>当前主题</h3>
            <ul class="this-page-menu">
              <li><a href="../_sources/documents/queue.txt">显示页面源码</a></li>
            </ul>
          <h3>快速搜索</h3>
            <form class="search" action="../search.html" method="get">
              <input type="text" name="q" size="18" /> <input type="submit" value="提交" />
              <input type="hidden" name="check_keywords" value="yes" />
              <input type="hidden" name="area" value="default" />
            </form>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related">
      <h3>导航</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="一般索引"
             accesskey="I">索引</a></li>
        <li class="right" >
          <a href="StringIO.html" title="PyMOTW: StringIO and cStringIO"
             accesskey="N">后一篇</a> |</li>
        <li class="right" >
          <a href="configparser.html" title="PyMOTW: ConfigParser"
             accesskey="P">前一篇</a> |</li>
        <li><a href="../index.html">PyMOTW v1.2 documentation</a> &raquo;</li> 
      </ul>
    </div>
    <div class="footer">
      &copy; <a href="../copyright.html">Copyright</a> 2008, vbarter &amp; liz.
      使用 <a href="http://sphinx.pocoo.org/">Sphinx</a> 0.5.1创建.
    </div>
  </body>
</html>